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Abstract 


In  this  paper  we  develop  a  theory  of  the  impact  of  behavioral  decision  making 
factors  on  the  evaluation  of  logistic  service  providers  under  performance-based  logistics 
and  provide  an  analysis  of  pilot  data  collected  in  an  attempt  to  find  support  for  that 
theory.  Based  on  a  review  of  the  logistic  measurement,  PBL,  and  behavioral  decision 
making  literature,  we  form  four  hypotheses  about  specific  impacts  of  process  measures 
and  variance  on  performance  evaluation  in  PBL.  Our  first  hypothesis  is  that  the 
difficulty  of  relating  component-level  measures  to  system-level  outcomes  will  lead  to  an 
increased  use  of  non-diagnostic  or  only  partially  diagnostic  process  measures.  We 
further  propose  that  these  process  measures  will  produce  a  dilution  effect  in  which 
system  outcomes  are  undervalued.  Our  third  hypothesis  is  that  absent  clear, 
observable  outcome  metrics  at  the  component  level,  decision  makers  will  increasingly 
rely  on  measures  of  inputs  as  surrogates  for  outputs.  Our  fourth  hypothesis  is  that 
absent  a  specific  guidance  on  how  to  value  variance,  decision  makers  will  tend  to 
overlook  this  important  component  of  performance.  We  report  results  from  a  pilot  test 
conducted  to  develop  an  instrument  that  will  be  used  to  try  to  find  support  for 
hypotheses  two  and  four. 

Keywords:  Performance-based  Logistics,  PBL,  Behavioral  Decision  Making, 
Process  and  Outcome,  Outcome  Bias 
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Executive  Summary 


Performance-Based  Logistics  (PBL)  is  an  initiative  that  the  Department  of 
Defense  (DoD)  has  targeted  for  ‘aggressive  implementation’  in  FY  2006-2009 
(Wolfowitz  2004).  It  is  an  initiative  intended  to  improve  weapon  system  logistics 
outcomes,  and  reduced  weapon  system  lifecycle  costs.  Provider  evaluation  in  PBL  is 
intended  to  center  on  clearly  specified  outcome  metrics,  and  mutually-agreed  upon 
goals  on  those  metrics  (DUSD-LMR  2001 )  with  the  idea  that  the  DoD  knows  best  what  it 
wants  in  terms  of  logistic  services,  but  the  vendor  may  know  best  how  to  provide  those 
services.  PBL  can  be  seen  as  an  extension  of  the  principle  of  “commander’s  intent”  in 
which  leadership  presents  goals,  but  subordinates  are  encouraged  to  choose  methods 
and  processes  (Apgar  IV  and  Keane  2004). 

Within  the  field  of  behavioral  decision  making,  there  is  a  substantial  literature 
which  shows  that  decision  makers  use  sub-optimal  heuristics  to  value  and  negotiate 
agreements  such  as  PBL  contracts  for  services,  and  are  subject  to  systematic  biases  in 
judgment  when  evaluating  performance  (Kahneman,  Slovic  et  al.  1982).  In  this  paper, 
we  draw  a  connection  between  the  intent  of  PBL  on  the  one  hand,  and  the  literature  on 
biases  and  heuristics  on  the  other.  Specifically,  we  review  the  literature  on  PBL  and 
logistics  service  measurement  as  it  relates  to  1 )  the  distinction  between  process  and 
outcome  measures  and  2)  the  significance  of  risk.  We  then  review  the  literature  on 
behavioral  decision  making  and  performance  evaluation  that  relates  to  those  same  two 
topics,  and  develop  hypotheses  regarding  1 )  the  potential  impact  of  process 
measurement  on  outcome  measurement,  and  2)  the  absence  of  stated  metrics  and 
goals  for  the  variance  (risk)  of  outcomes.  We  develop  specific,  testable  hypotheses 
from  this  review  of  the  literature,  report  on  a  test  of  these  hypotheses  in  a  laboratory 
experiment,  and  discuss  the  implications  of  our  findings  on  practice. 

PBL  is  an  evolving  concept  within  the  DoD,  and  clarification  on  the  metrics  which 
should  be  used  to  assess  weapon  system  logistics  outcomes  has  been  recently  issued 
which  emphasizes  that  system-level  outcomes  such  as  operational  availability  should 
be  used  to  evaluate  PBL  candidates  and  the  performance  of  PBL  providers  (Wynne 
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2004).  The  system-level  emphasis  of  this  clarification  is  significant  and  proper,  as 
warfighting  outcomes  are  clearly  only  impacted  by  system  level  (as  opposed  to 
component  level)  performance. 

But  PBL  is  still  being  applied  at  the  component  level  and  there  is  no  clear 
guidance  (to  our  knowledge)  on  how  to  link  component  level  variables  like  time-to- 
failure  to  system  level  outcomes  like  operational  availability.  Indeed,  a  recent 
Government  Accountability  Office  (GAO)  report  found  that  most  of  the  185  PBL 
contracts  they  were  able  to  identify  in  the  DoD  were  written  at  the  component  or 
subsystem  level,  and  they  suggested  that  contracting  at  the  component  level  should 
continue  to  be  preferred  to  contracting  at  the  system  level  (GAO  2004). 

Even  in  the  private  sector,  the  measurement  and  performance  assessment  of 
logistic  services  is  known  to  be  a  difficult  task.  Proper  valuation  of  the  outcomes  of 
logistic  services  (as  opposed  to  merely  valuing  inputs,  such  as  cost)  must  include  some 
assessment  of  difficult  to  quantify  factors  such  as  customer  satisfaction,  and  risk 
reduction  (Lambert  and  Burduroglu  2000).  This  outcome  measurement  problem  is 
made  more  difficult  because  so  many  of  the  traditional  logistics  measures  are  process 
measures  (Caplice  and  Sheffi  1994).  Nor  is  it  easier  when  the  services  are  provided  to 
the  DoD,  where  outcomes  cannot  be  reduced  to  measurable  quantities  like  profit  or 
shareholder  value  (Camm,  Blickstein  etal.  2004). 

The  DoD  of  course  has  no  simple  overarching  valuation  metric  such  as  profit, 
and  it  has  no  simple  revenue  surrogates.  Valued  outcomes  have  to  do  with  military 
missions;  thus,  even  if  logistic  services  for  a  weapon  system  are  provided  at  an 
aggregate  level  by  a  single  provider,  they  are  difficult  to  value  and  price.  At  the  level  of 
a  subassembly  or  single  logistic  element,  the  problem  is  compounded.  Unless  decision 
makers  have  comprehensive  models  of  weapon  systems  logistics,  in  which  the 
important  performance  dimensions  of  all  critical  components  are  modeled,  they  cannot 
value  a  component-level  contract  in  terms  of  system  level  outcomes  like  operational 
availability.  Such  models  have  not  been  required,  and  we  have  no  evidence  that  they 
are  being  used  in  the  field. 
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In  situations  without  clearly  observable  outcome  measures  and  valuation 
functions,  decision  makers  are  known  to  place  a  heavy  weight  on  surrogates  such  as 
process  measures,  or  even  input  measures  (Chinander  and  Schweitzer  2003).  Some  of 
these  may  not  correlate  well  with  system-level  outcomes.  Under  PBL,  decision  makers 
must  determine  relevant  outcomes  for  component  level  contracts  and  separate 
diagnostic  measures  (those  that  correlate  well  with  desired  system  outcomes)  from  non¬ 
diagnostic  ones.  However,  decision  makers  are  known  to  pursue  information  even 
when  it  is  non-diagnostic  and  non-instrumental  (knowledge  of  the  measure  would  not  or 
should  not  change  decisions).  Unfortunately,  once  obtained,  such  non-instrumental 
information  may  be  treated  as  if  it  were  instrumental  (Bastardi  and  Shafir  1998).  That 
is,  decision  makers  pursue  information  they  do  not  need,  then  act  upon  it.  In  our  paper, 
we  investigate  this  tendency  in  decision  makers  asked  to  evaluate  provider  performance 
under  a  hypothetical  PBL  contract. 

It  might  be  claimed  that  additional  information  could  never  hurt  the  decision 
process  (aside  perhaps  from  the  cost  of  gathering  it)  but  at  least  two  sets  of  research 
findings  indicate  that  such  confidence  would  be  misplaced.  The  curse  of  knowledge  is 
a  dysfunctional  decision  making  pattern  that  occurs  when  a  decision  maker  knows 
information  that  they  would  be  better  off  to  ignore,  but  they  cannot  ignore  it  (Camerer, 
Loewenstein  et  al.  1989).  The  classic  example  is  a  wine  merchant,  who  over  prices  his 
good  wine,  and  under  prices  his  bad  wine  and  thus  loses  revenue  on  both  sides  from 
customers  who  do  not  know  as  much  about  wine  as  he  does.  In  our  case,  the  decision 
maker  who  pursues  non-diagnostic  process  information  may  misestimate  provider 
performance  because  of  it.  A  related  bias  is  the  dilution  effect:  the  tendency  for  non¬ 
diagnostic  information  to  cause  diagnostic  information  to  be  undervalued  (Nisbett, 

Zukier  et  al.  1981 ).  In  the  case  of  PBL,  if  a  decision  maker  captures  process  metrics, 
he  or  she  may  not  be  able  to  place  them  in  the  proper  context  relative  to  a  system  level 
outcome,  and  the  impact  of  an  important  outcome  metric  may  be  diluted.  In  our  paper, 
we  investigate  the  tendency  of  decision  makers  to  dilute  system  outcomes  when  given 
knowledge  of  process  variables. 
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A  special  case  of  the  misuse  of  non-diagnostic  information  is  the  use  of 
information  about  inputs.  The  input  bias  is  the  tendency  to  make  judgments  about  the 
quality  of  outcomes  based  on  the  value  of  inputs  (Chinander  and  Schweitzer  2003).  For 
example,  people  tend  to  judge  the  quality  of  a  product  or  service  higher  when  they  have 
to  wait  longer  for  it  (Maister  1985).  This  bias  is  thought  to  play  an  especially  significant 
role  in  evaluation  when  outcomes  are  difficult  to  observe  or  measure.  In  the  case  of 
PBL  contracts,  the  evaluation  of  proposals  based  solely  on  the  relative  cost  of 
alternatives  would  be  an  example  of  an  input  bias.  Also,  a  performance  evaluation  that 
considered  investments  a  provider  made  in  achieving  outcomes  would  be  an  example 
of  an  input  bias.  In  our  paper,  we  investigate  the  susceptibility  of  decision  makers  to  an 
input  bias  when  evaluating  the  performance  of  logistic  service  providers. 

There  are  other  reasons  why  decision  makers  may  seek  out  component-level 
process  measures,  even  when  they  have  been  directed  to  look  at  system  level  outcome 
measures.  Process  measures  allow  a  better  degree  of  control  over  the  internal 
workings  of  a  process.  They  may  not  reduce  uncertainty  around  outcomes,  but  they  do 
give  decision  makers  a  sense  that  outcomes  are  more  directly  under  their  control.  Risk 
preferences  vary  widely,  but  in  addition  to  individual  differences  in  risk  aversion  or  risk 
seeking  behavior,  decision  makers  tend  to  prefer  controllable  to  uncontrollable  ones 
even  to  the  extent  that  they  will  maintain  illusions  about  the  degree  of  control  they  have 
over  a  situation  (Langer  1975).  The  preference  for  controllable  risks  is  said  in  part  to  be 
related  to  a  general  bias  decision  makers  have  that  their  own  abilities  are  better  than 
others  (Howell  1971).  Of  course,  part  of  the  logic  of  performance-based  outsourcing  is 
that  providers  are  more  capable  of  dealing  with  the  internal  processes  of  the  logistics 
service.  But  decision  makers  appear  to  maintain  this  preference  for  controllable  risks, 
and  to  support  their  bias  toward  exaggerated  self-assessments,  even  when  they  would 
be  better  off  with  less  control  (Klein  and  Kunda  1994). 

In  delegating  the  decisions  on  how  to  accomplish  outcome  goals  to  a  provider, 
programs  seek  to  use  PBL  to  transfer  some  of  the  process  and  financial  risk  of  the 
logistic  service  to  the  provider;  in  contracting  to  deliver  outcomes  while  assuming 
responsibility  for  processes,  providers  accept  that  risk  at  a  specified  price.  The 
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assessment  of  these  risks  is  part  of  a  business  case  analysis  required  for  every 
implementation  of  PBL  in  the  Navy  (Young  2003).  To  our  knowledge  however,  DoD- 
level  PBL  guidance  does  not  require  any  specific  measures  of  outcome  risk,  or  process 
risk  transfer. 

The  biases  and  heuristics  literature  makes  it  clear  that  human  decision  makers 
are  poor  intuitive  statisticians  (Kahneman,  Slovic  et  al.  1982).  However,  as  Cohen 
(1982)  pointed  out,  if  decision  makers  could  intuitively  grasp  statistical  concepts,  what 
would  be  the  point  of  offering  classes  in  them?  Whether  it  is  a  question  of  education  or 
irrationality,  it  seems  clear  that  most  decision  makers  do  not  have  an  intuitive  model  that 
allows  them  to  value  variance  in,  for  example,  operational  availability.  In  our  paper,  we 
investigate  the  tendency  for  decision  makers,  even  when  trained  in  risk  assessment,  to 
undervalue  the  impact  of  outcome  variance. 

The  investigations  in  our  paper  are  all  made  through  laboratory  experiments: 
questionnaires  asking  decision  makers  to  evaluate  PBL  scenarios.  The  results  have 
only  limited  generalizability  to  the  actual  management  of  extant  PBL  contracts,  or  to  the 
valuation  and  pricing  of  PBL  contracts.  However,  the  results  do  have  implications  for 
the  continued  evolution  of  PBL,  and  the  need  for  greater  specificity  in  guidance.  That  is, 
if  decision  makers  under  PBL  are  subject  to  the  same  limitations  as  decision  makers  in 
our  study,  it  indicates  the  need  for  the  DoD  to  develop  specific  guidance  with  regard  to 
risk  measurement  and  valuation,  and  to  require  comprehensive  system-level  models  to 
value  and  price  component  level  contracts,  and  evaluate  component  level  logistic 
service  provider  performance. 
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Introduction 


Performance-Based  Logistics  (PBL)  is  an  initiative  that  the  Department  of 
Defense  (DoD)  has  targeted  for  ’’aggressive  implementation”  in  FY  2006-2009 
(Wolfowitz,  2004).  It  is  an  initiative  intended  to  improve  weapon  system  logistics 
outcomes,  and  reduce  weapon  system  lifecycle  costs.  Important  weapon  system 
logistics  outcomes  have  been  defined  (each  of  the  following  will  be  explained  in  more 
detail  below)  as  operational  availability,  operational  reliability,  cost  per  usage,  logistics 
footprint,  and  logistics  response  time  (Wynne,  2004). 

PBL  is  different  from  traditional  military  logistics  management  in  that  program 
managers  are  supposed  to  dictate  what  a  logistics  service  provider  should  deliver  in 
terms  of  measurable  outcomes,  not  how  the  delivery  of  those  outcomes  should  be 
accomplished.  It  can  be  seen  as  an  extension  of  the  principle  of  “commander’s  intent” 
in  which  leadership  presents  goals,  but  subordinates  are  encouraged  to  choose 
methods  and  processes  (Apgar  &  Keane,  2004).  Whether  the  service  provider  is  an 
organic  DoD  organization  or  a  private  sector  organization,  it  must  enter  into  a 
contractual  relationship  with  weapon  system  managers  in  which  payment  is  based  on 
performance  relative  to  agreed-upon  outcome  goals. 

In  delegating  the  decisions  on  how  to  accomplish  outcome  goals  to  a  vendor, 
programs  seek  to  use  PBL  to  transfer  some  of  the  operational  and  financial  risk  of  the 
methods  and  processes  used  to  the  vendor;  and  in  contracting  to  deliver  outcomes 
while  assuming  responsibility  for  processes,  vendors  accept  that  risk  at  a  specified 
price.  The  assessment  of  these  risks  is  part  of  a  business  case  analysis  required  for 
every  implementation  of  PBL  in  the  Navy  (Young,  2003). 

The  measurement  and  performance  assessment  of  logistic  services  is  clearly  a 
difficult  task.  Cost  reduction  is  not  an  adequate  assessment  of  the  strategic  benefits  of 
logistic  services,  which  must  include  some  assessment  of  difficult-to-quantify  factors 
such  as  customer  satisfaction  and  risk  reduction  (Lambert  &  Burduroglu,  2000).  This 
measurement  problem  is  not  made  easier  when  an  organization  focuses  on  outcomes 
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to  be  acquired — rather  than  processes  to  be  monitored — because  so  many  of  the 
traditional  logistics  measures  are  process  measures  (Caplice  &  Sheffi,  1994).  Nor  is  it 
easier  when  the  services  are  provided  to  the  DoD,  where  outcomes  cannot  be  valued  in 
terms  of  profit  or  shareholder  value  (Camm,  Blickstein  et  al.,  2004). 

Indeed,  it  is  one  of  the  premises  of  this  paper  that  because  logistics  is  so  often 
considered  an  internal  process  (or  even  a  sub-process)  of  an  activity,  outcomes  are 
difficult  to  value.  This  is  especially  true  for  weapon  systems  logistics,  in  which  even 
what  are  usually  considered  outcome  measures  (e.g.,  operational  availability)  have  no 
one-to-one  correspondence  to  recognizable  outcomes  valued  by  the  DoD  (e.g.,  mission 
success).  Without  clearly  observable  (let  alone  valuable)  outcome  measures,  decision 
makers  are  known  to  place  a  heavy  weight  on  surrogates  such  as  process  measures,  or 
even  input  measures  (Chinander  &  Schweitzer,  2003).  We  will  investigate  the  potential 
for  bias  in  performance  evaluation  created  by  the  attempt  to  measure  and  value  system- 
level  outcomes,  when  the  services  contracted  are  only  sub-processes  of  weapon 
system  operations  and  support. 

Another  premise  of  this  paper  is  that  the  risk  transfer  intended  by  delegating 
processes  must  be  assessed  through  the  measurement  of  outcome  variance.  After  all, 
risk,  in  Finance  or  Logistics,  is  synonymous  with  variance.  In  allowing  a  vendor  to 
choose  methods  and  processes,  a  manager  must  hold  him  or  her  accountable  for  the 
reliable  delivery  of  outcomes.  One  DoD  organization,  in  issuing  guidance  for  vendor 
management  under  PBL,  stated  the  following: 

Minimal  contract  management  involvement  is  anticipated  as  long  as  the 
contractor  meets  contractually  specified  performance  metrics.  However,  our 
involvement  may  increase  if  the  contractor  systems  and  processes  are  not 
functioning  correctly  and  end  users  are  not  appropriately  supported.  (Bogusz, 
Taylor  et  al.,  2002) 

In  our  opinion,  true  risk  transfer  has  not  taken  place  if  ’’involvement  may 
increase”  when  contractor  processes  are  not  functioning  correctly.  If  the  vendor  is  not 
responsible  for  correcting  the  variance  in  their  processes,  then  the  DoD  is  still  in  the 


2 


business  of  managing  processes,  not  outcomes,  and  should  continue  to  measure 
processes  as  well  as  outcomes  (and  should  not  be  paying  someone  else  for  the 
management  of  the  process).  This  is  not  to  say  that  a  manager  should  simply  ignore 
the  potential  impact  of  process  variance.  But  consequential  process  variance  will  show 
up  in  outcome  variance  as  well,  and  outcome  variance  is  itself  an  important 
performance  dimension. 

For  example,  if  a  vendor  has  contracted  to  provide  a  certain  level  of  operational 
availability  for  a  component,  one  of  the  sub-processes  that  will  determine  operational 
availability  is  maintenance  lead  time.  Problems  (variance)  in  the  sub-process  of 
maintenance  lead  time  will  create  variance  in  operational  availability.  That  is,  one  does 
not  need  to  monitor  a  sub-process  to  observe  the  impact  of  the  variance  of  that  sub¬ 
process:  it  can  be  observed  through  variance  in  system-level  outcomes.  Of  course, 
variance  in  operational  availability  is  itself  an  important  performance  dimension,  as  it 
determines  the  risk  that  the  number  of  mission-capable  assets  will  fall  below  some 
planning  threshold.  But  in  addition  to  determining  direct  performance  implications  of 
outcome  variance,  by  monitoring  outcome  variance,  the  DoD  implicitly  monitors 
variance  (risk)  in  all  of  the  sub-processes  that  determine  that  outcome.  Risk  can  be 
said  to  have  been  successfully  transferred  only  when  operational  (or  financial)  risk  has 
been  reduced. 

In  the  next  section,  we  will  review  the  literature  on  PBL  and  logistics  service 
measurement  as  it  relates  to:  1 )  the  distinction  between  process  and  outcome 
measures  and  2)  the  significance  of  risk.  We  will  then  review  the  literature  on 
behavioral  decision  making  and  performance  evaluation  that  relates  to  those  same  two 
topics,  and  develop  hypotheses  regarding:  1)  the  potential  impact  of  process 
measurement  on  outcome  measurement  and  2)  the  absence  of  stated  metrics  and 
goals  for  the  variance  of  outcomes.  We  will  develop  specific,  testable  hypotheses  from 
this  review  of  the  literature.  Next,  we  will  report  on  preliminary  findings  relating  to  the 
development  of  an  instrument  to  test  two  of  our  hypotheses.  Finally,  we  will  discuss  the 
implications  of  our  hypotheses  and  managerial  implications,  should  support  for  them  be 
found  in  later  work. 
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PBL,  Logistics  Measurement  and  Behavioral 
Decision-Making  Research 


The  key  word  in  Performance-Based  Logistics  is  “performance,”  which  means 
“specification  and  valuation  of  outcome  rather  than  process.”  There  are  a  number  of 
other  related  US  Government  and  DoD  initiatives  which  use  the  word  performance  in 
this  way — for  example,  Performance-Based  Service  Contracting  (OFPP,  1998)  and 
Performance-Based  Service  Acquisition  (DUSD-DAR,  2000).  The  use  of  the  term 
performance  in  all  of  these  initiatives  is  at  least  in  part  intended  to  imply  a  break  from 
the  past,  which  is  seen  as  involving  needlessly  complex  specification  and  micro- 
management  of  vendor  processes,  when  vendors  sometimes  understand  the  methods 
of  service  delivery  better  than  the  governmental  customer  who  is  buying  the  service. 
Vendor  evaluation  in  “performance”  initiatives  is  intended  to  center  on  clearly  specified 
outcome  metrics  and  mutually-agreed  upon  goals  on  those  metrics  (DUSD-LMR,  2001), 
with  the  idea  that  the  DoD  knows  best  what  it  wants  in  terms  of  logistic  services,  but  the 
vendor  may  know  best  how  to  provide  those  services. 

Logistic  services  present  a  problem  in  this  regard,  however,  because  many 
aspects  of  performance — especially  those  relating  to  the  benefits  (and  not  the  costs)  of 
the  service — deal  with  difficult-to-measure  factors  such  as  customer  perception  of 
service  quality  (Mentzer,  Flint  et  al.,  1999)  or  customer  value  added  (Lambert  & 
Burduroglu,  2000).  Also,  the  vast  majority  of  logistic  services  measures  involve  what 
have  been  called  “utilization”  measures  (ratios  of  actual  to  normalized  inputs)  or 
“productivity”  measures  (ratios  of  actual  output  to  actual  input)  on  single  resources — in 
other  words,  sub-processes  within  the  logistic  service,  such  as  items  picked/man-hour 
in  a  warehouse.  Few  total  factor  “effectiveness”  measures  (ratios  of  outcomes  to 
normalized  outcomes,  or  goals)  are  used  (Caplice  &  Sheffi,  1994).  But  these  are 
precisely  the  sorts  of  measures  prescribed  under  PBL. 

Weapon  system  logistics  may  be  even  more  difficult  to  assess  in  terms  of 
outcomes  because  the  relationship  between  support  services  and  mission  outcomes 
valued  by  the  DoD  is  even  more  attenuated  than  the  relationship  between  support 
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services  and  profit  in  the  commercial  sector  (Doerr,  Lewis  et  al. ,  in  press).  Finally,  PBL 
has  been  implemented  for  individual  logistic  elements  (e.g.,  inventory  alone  or  repair 
alone)  at  the  component  or  sub-assembly  level  of  weapons  systems.  The  relationship 
between  logistic  support  services  at  the  component  or  individual  logistic-element  level, 
and  outcomes  at  the  weapon-system  level  is  difficult  to  determine  and  probabilistic  in 
nature,  as  it  depends  on  the  performance  of  the  other  critical  components  and  logistical 
elements  that  determine  overall  system  performance. 

In  the  past,  measures  have  been  recommended  for  PBL  contracts  that  are 
clearly  process,  and  not  outcome  measures.  For  example,  Fill  Rate  or  back-order  aging 
rates  have  been  suggested  (Bogusz,  Taylor  et  al.,  2002).  When  measured  at  a 
component  level  (e.g.,  fill  rate  for  a  fuel  cell),  the  connection  of  these  measurements  to 
system-level  outcomes  is  so  tenuous  as  to  be  impossible  to  determine  in  isolation  from 
the  process  metrics  of  other  major  components  (Kang,  Doerr  et  al.,  2005).  In  recent 
high-level  guidance  for  PBL,  the  DoD  has  made  it  clear  that  system-level  performance 
(outcome)  metrics  should  be  negotiated  with  a  PBL  vendor.  A  recent  memo  from  the 
Deputy  Undersecretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (Wynne, 
2004)  listed  five  key  performance  criteria:  1)  weapon  system  operational  availability,  2) 
weapon  system  operational  reliability,  3)  weapon  system  cost  per  usage,  4)  logistics 
footprint  for  a  weapon  system,  and  5)  response  time  required  for  weapon  system 
logistics  support.  However,  even  in  this  reduced  subset  of  important  variables,  aspects 
of  reliability  (time  to  failure  for  a  component)  and  logistics  footprint  (spares  inventory 
levels  in  the  field)  can  be  seen  as  process  variables  which  help  to  determine  the 
’’outcome”  of  operational  availability. 

Moreover,  PBL  is  still  being  applied  at  the  subsystem,  or  major  assembly  level, 
and  there  is  no  clear  guidance  on  how  to  link  component-level  variables  like  time-to- 
failure  to  system-level  outcomes  like  operational  availability.  Indeed,  a  recent  GAO 
report  found  that  most  of  the  185  PBL  contracts  they  were  able  to  identify  in  the  DoD 
were  written  at  the  component  or  subsystem  level,  and  they  suggested  that  contracting 
at  that  level  should  be  preferred  to  contracting  at  the  platform,  or  system  level  (GAO, 
2004).  This  GAO  recommendation  seems,  in  large  part,  to  have  been  justified  on  the 
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basis  of  commercial  practices;  nowhere  does  it  address  the  significant  difference 
between  DoD  and  private-sector  logistics  in  terms  of  relating  component-level  variables 
such  as  component  reliability  or  availability  to  system-level  outcomes.  In  the  private 
sector,  the  central  financial  valuation  question  in  such  a  contract  (i.e.,  the  question  of 
how  to  price  it — apart  from  any  strategic  consideration)  would  revolve  around  the  cost 
and  revenue  implications:  if  the  rate  of  return  of  the  contract  was  predicted  to  exceed  a 
corporate  hurdle  rate  for  profitability,  it  would  likely  be  adopted  (depending,  in  addition 
on  the  strategic  considerations). 

The  DoD,  of  course,  has  no  simple  valuation  metric  such  as  profit,  and  it  has  no 
simple  revenue  surrogates.  Valued  outcomes  have  to  do  with  military  missions,  and  the 
driving  surrogate  for  those  (from  the  logistics  point  of  view)  is  readiness.  And  while 
readiness  as  a  surrogate  may  be  seen  to  beg  the  question  of  valuation  (e.g.,  how  ready 
for  what  mission  contingency?),  its  common  operationalization  as  Ao  (operational 
availability)  is  even  more  distal.  Thus,  even  if  logistic  services  for  a  weapon  system  are 
provided  at  an  aggregate  level  by  a  single  provider,  they  are  difficult  to  value  and  price. 
At  the  level  of  a  subassembly  or  single  logistic  element,  the  problem  is  compounded. 
Reducing  mean  time  to  repair  for  an  auxiliary  power  unit  (APU),  for  example,  may 
increase  the  probability  that  a  spare  APU  will  be  on  the  shelf  if  needed  (though  even 
that  relationship  is  not  trivial  to  derive);  but,  the  impact  of  this  higher  probability  on  Ao  of 
an  aircraft  depends  on  the  sparing  level,  failure  rates,  and  time  to  repair  of  all  the  other 
critical  components  of  the  aircraft.  Unless  decision  makers  have  comprehensive 
models  of  weapons  systems  logistics,  in  which  the  important  performance  dimensions 
of  all  critical  components  are  modeled,  they  cannot  value  a  component-level  contract  in 
terms  of  system-level  outcome  surrogates  like  Ao.  Such  models  have  not  been 
required,  and  we  have  no  evidence  that  they  are  being  used  in  the  field. 

So,  if  component-level  contracts  are  not  being  valued  in  terms  of  system-level 
outcomes,  how  are  new  contracts  being  valued  and  priced?  And  how  is  performance 
on  ongoing  component-level  contracts  being  evaluated?  This,  of  course,  is  an  empirical 
question,  and  we  have  no  field  evidence  to  address  this  question  one  way  or  the  other. 
However,  we  can  draw  on  a  substantial  literature  in  the  domain  of  behavioral  decision 
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making  to  build  hypotheses  about  how  decision  makers  typically  value  and  evaluate 
performance  when  outcomes  are  difficult  to  observe  or  measure. 

There  is  a  substantial  literature  to  suggest  that  decision  makers  (even  in 
unambiguous  situations)  use  sub-optimal  heuristics  to  solve  problems  and  are  subject 
to  biases  in  judgment  (Kahneman,  Slovic  et  al.,  1982).  In  PBL,  we  think  measurement 
and  valuation  difficulties  will  promote  certain  biases  and  cause  decision  makers  to 
overlook  important  aspects  of  valuation  (risk).  These  difficulties  include  the  distal 
relationship  between  component-level  processes  and  system-level  outcomes,  and  the 
difficulty  of  proper  valuation  of  variance  in  outcome  measures. 

There  has  been  research  which  suggests  that  performance-based  evaluation 
may  itself  create  judgmental  bias.  The  outcome  bias  is  the  tendency  to  assume  that 
outcomes  and  processes  are  more  strongly  related  than  they  really  are — that  correct 
process  decisions  lead  to  desired  outcomes,  even  when  they  do  not  (Baron  &  Hershey, 
1988).  Because  good  decisions  may  sometimes  lead  to  bad  outcomes  (and  vice- 
versa),  some  degree  of  evaluation  based  on  processes  themselves  seems  normatively 
preferable.  There  has  been  some  debate  about  whether  judgment  based  solely  on 
outcome  might  be  justified  at  all  (Hershey  &  Baron,  1992;  Lipshitz,  1995);  yet,  a 
situation  in  which  the  decision  maker  knows  relatively  little  about  the  processes  used  to 
obtain  the  outcome  has  been  said  to  be  indicative  of  a  case  where  judgment  by 
outcome  might  be  preferred. 

Even  taking  the  premise  of  judgment-by-outcome  as  given  for  PBL,  there 
remains  the  problem  of  determining  relevant  outcomes  for  component-level  contracts 
and  separating  diagnostic  measures  (those  that  correlate  well  with  desired  system 
outcomes)  from  non-diagnostic  ones.  Although  guidance  has  been  given  as  to  the 
nature  of  system-level  outcome  measures  that  should  be  included  in  PBL  contracts 
(Wynne,  2004),  there  has  been  no  clear  guidance  (to  our  knowledge)  on  how  to  relate 
sub-system  processes  and  outcomes  to  system  outcomes,  and  no  strict  guidance  that 
managers  should  limit  evaluation  to  those  metrics  that  relate  to  system-level  outcomes. 
In  situations  with  uncertainty,  decision  makers  are  known  to  pursue  information  even 
when  it  is  non-diagnostic  and  non-instrumental  (knowledge  of  the  measure  would  not  or 
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should  not  change  decisions).  Unfortunately,  once  obtained,  such  non-instrumental 
information  may  be  treated  as  if  it  were  instrumental  (Bastardi  &  Shafir,  1998).  That  is, 
decision  makers  pursue  information  they  do  not  need,  then  act  upon  it. 

Logistic  services,  as  already  noted,  are  difficult  to  measure,  and  a  large  number 
of  process  metrics  have  been  used  for  various  logistics  elements  (Caplice  &  Sheffi, 
1994).  When  applied  at  a  component  level,  many  of  these  metrics  will  correlate  at  least 
to  some  degree  with  system-level  outcomes  such  as  operational  availability  (though 
some  will  not).  But  the  precise  nature  of  the  relationship  between  a  logistics  measure 
such  as  warehouse  turnover  for  a  certain  type  of  tire,  and  operational  availability  of  a  jet 
is  quite  difficult  to  determine.  Given  the  abundance  of  such  measures,  however,  and 
the  difficulty  of  the  task  of  measuring  system  outcomes,  we  think  it  is  likely  that  decision 
makers  will  pursue  such  non-diagnostic,  or  partially  diagnostic  information  about  the 
logistic  processes  of  components  under  PBL  contracts. 

Hypothesis  1:  In  evaluating  vendor  performance  for  outsourced  logistics 
services,  decision  makers  will  seek  out  process  measures  that  only  partially  correlate 
with  system-level  outcomes,  even  when  they  are  given  system  outcome  measurements. 

This  would  not  necessarily  be  a  problem  (indeed,  according  to  the  tenets  of  the 
outcome  bias,  it  might  be  beneficial)  were  it  not  for  the  very  quantity  of  process 
measures  available,  and  the  difficulty  of  discerning  those  that  might  be  the  most 
instrumental  from  those  likely  to  be  the  least.  It  might  be  claimed  that  additional 
information  could  never  hurt  the  decision  process  (aside  perhaps  from  the  cost  of 
gathering  it),  but  at  least  two  sets  of  research  findings  indicate  that  such  confidence 
would  be  misplaced. 

The  curse  of  knowledge  is  a  dysfunctional  decision-making  pattern  that  occurs 
when  decision  makers  know  information  they  would  be  better  off  to  ignore,  but  once 
they  know  it,  they  cannot  ignore  it  (Camerer,  Loewenstein  et  al. ,  1989).  The  classic 
example  is  a  wine  merchant  who  over-prices  his  good  wine  and  under-prices  his  bad 
wine  because  he  “knows”  about  his  wine.  He,  thus,  loses  revenue  on  both  sides  from 
customers  who  do  not  know  as  much  about  wine  as  he  does.  In  our  case,  the  decision 
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maker  who  pursues  non-diagnostic  process  information  may  tend  to  overvalue  vendor 
performance  (if  the  non-diagnostic  numbers  were  better  than  the  system  outcome 
metrics)  or  undervalue  it  (if  the  non-diagnostic  numbers  were  worse  than  the  system 
outcome  metrics). 

Another  related  bias  is  the  dilution  effect:  the  tendency  for  large  quantities  of 
non-diagnostic  information  to  cause  diagnostic  information  to  be  undervalued  (Nisbett, 
Zukier  et  al.,  1981 ).  In  the  case  of  PBL,  if  a  decision  maker  captures  large  numbers  of 
logistic  process  metrics,  the  impact  of  an  important  outcome  metric  may  be  overlooked. 

Hypothesis  2:  Decision  makers  given  non-diagnostic  or  partially  diagnostic 
information  about  component-level  processes  will  use  it  to  moderate  their  evaluations  of 
system-level  outcomes. 

A  special  case  of  the  misuse  of  non-diagnostic  information  is  the  use  of 
information  about  inputs.  The  input  bias  is  the  tendency  to  make  judgments  about  the 
quality  of  outcomes  based  on  the  value  of  inputs  (Chinander  &  Schweitzer,  2003).  For 
example,  people  tend  to  judge  the  quality  of  a  product  or  service  higher  when  they  have 
to  wait  longer  for  it  (Maister,  1985).  This  bias  is  thought  to  play  an  especially  significant 
role  in  evaluation  when  outcomes  are  difficult  to  observe  or  measure.  In  the  case  of 
PBL  contracts,  the  evaluation  of  proposals  based  solely  on  the  relative  cost  of 
alternatives  would  be  an  example  of  an  input  bias.  Also,  a  performance  evaluation  that 
considered  investments  a  vendor  made  in  achieving  outcomes  would  be  an  example  of 
an  input  bias. 

Hypothesis  3:  Decision  makers  will  use  input  information  such  as  cost  or  effort 
expended  on  component-level  processes  to  moderate  their  judgments  of  system-level 
outcomes. 

There  are  other  reasons  why  decision  makers  may  seek  out  component-level 
process  measures,  even  when  they  have  been  directed  to  use  system-level  outcome 
measures.  Process  measures  allow  a  better  degree  of  control  over  the  internal 
workings  of  a  process.  They  may  not  reduce  uncertainty  around  outcomes,  but  they  do 
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give  decision  makers  a  sense  that  outcomes  are  more  directly  under  their  control. 

Under  PBL,  decision  makers  are  supposed  to  avoid  the  temptation  to  exercise  process 
control,  and  specify  and  evaluate  contractors  based  solely  on  outcomes.  However,  this 
intention  seems  hard  to  realize.  When  we  first  presented  data  we  had  gathered  that 
indicated  some  stakeholders  in  the  PBL  process  felt  it  entailed  ’’too  many  metrics,”  a 
DoD  decision  maker  raised  the  objection  that  the  number  of  metrics  was  irrelevant,  and 
pointed  out  that  pilots  have  a  superabundance  of  indicators  in  their  cockpits,  most  of 
which  they  ignore  unless  something  is  going  wrong.  But  of  course,  under  performance- 
based  contracting,  one  isn’t  supposed  to  be  flying  the  plane:  one  is  supposed  to  be 
buying  a  ticket  to  ride  as  a  passenger. 

Risk  preferences  vary  widely,  but  in  addition  to  individual  differences  in  risk- 
aversion  or  risk-seeking  behavior,  decision  makers  tend  to  prefer  controllable  to 
uncontrollable  risk  even  to  the  extent  that  they  will  maintain  illusions  about  the  degree  of 
control  they  have  over  a  situation  (Langer,  1975).  The  preference  for  controllable  risks 
is  said,  in  part,  to  be  related  to  a  general  bias  decision  makers  have  that  their  own 
abilities  are  better  than  others’  (Howell,  1971).  Of  course,  part  of  the  logic  of 
performance-based  outsourcing  is  that  vendors  are  more  capable  of  dealing  with  the 
internal  processes  of  the  service.  But  decision  makers  appear  to  maintain  a  preference 
for  controllable  risks,  and  to  support  their  bias  toward  exaggerated  self-assessments, 
even  when  they  would  be  better  off  with  less  control  (Klein  &  Kunda,  1994). 

Finally,  as  already  noted,  we  think  the  variance  (risk)  associated  with  outcomes 
is  itself  an  important  performance  measure  that  should  be  explicitly  considered  in 
valuing  and  evaluating  outsourced  logistic  services.  This  is  both  because  (process)  risk 
transfer  is  an  intended  outcome  of  PBL  (and,  hence,  it  should  be  measured  to  see  if  that 
outcome  is  obtained),  but  also  because  variance  in  outcomes  directly  affects 
contingency  planning  for  operations.  Unfortunately,  while  PBL  guidance  clearly 
indicates  the  importance  of  risk  (Young,  2003),  that  guidance  does  not  require  specific 
measures  of  risk.  The  biases  and  heuristics  literature  makes  it  clear  that  human 
decision  makers  are  poor  intuitive  statisticians  (Kahneman,  Slovic  et  al.,  1982).  Indeed, 
one  of  the  early  criticisms  of  the  research  on  biases  and  heuristics  is  that  in  part,  it 


merely  represents  a  set  of  tests  of  intelligence  or  educational  achievement  (Cohen, 
1982):  if  decision  makers  could  intuitively  grasp  statistical  concepts,  what  would  be  the 
point  of  offering  classes  in  them?  It  would  be  beyond  the  scope  of  this  paper  to 
document  the  many  intuitive  heuristics  decision  makers  use  to  deal  with  uncertainty  and 
risk.  Whether  it  is  a  question  of  education  or  irrationality,  however,  it  seems  clear  that 
most  decision  makers  do  not  have  an  intuitive  model  that  would  allow  them  to  value 
variance  in,  for  example,  operational  availability. 

Hypothesis  4:  Decision  Makers  will  undervalue  or  ignore  information  about 
outcome  variance,  unless  it  is  made  salient  to  them,  along  with  a  method  for  its  proper 
valuation. 
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Scale  Development  and  Results  of  a  Pilot  Experiment 


In  this  section  we  describe  both  the  results  of  a  scale  development  effort  to 
measure  vendor  performance  and  pilot  results  from  data  collected  to  help  validate  the 
scale.  Findings  are  also  given  from  these  pilot  data  that  relate  to  two  of  our  hypotheses. 

Participants. 

Participants  were  63  professional  military  officers  enrolled  in  an  MBA  program  at 
a  public  university.  The  average  participant  age  was  33.6  years  with  an  average  of  1 1 .7 
years  of  active  duty  service;  86.7  percent  of  the  participants  were  male.  Participants 
had  an  average  of  4.3  years  of  experience  in  logistics  and  1 .3  years  of  experience  in 
contract  management.  66.7  percent  of  the  participants  indicated  prior  knowledge  of 
PBL. 

Protocol. 

Participants  were  given  a  short  description  of  a  scenario  in  which  the  depot-level 
support  of  a  weapon  system  component  was  outsourced  to  a  vendor.  Participants  were 
told  that  the  component,  weapon  system  and  vendor  were  fictitious.  They  were  told  that 
their  responses  were  to  be  used  to  assess  biases  based  on  variance  in  decision 
scenarios.  Each  participant  was  given  one  of  three  scenarios,  as  described  below.  No 
extra  credit  was  given  for  participation  in  the  study,  and  participants  were  assured  both 
of  the  confidentiality  of  their  responses  and  the  voluntary  nature  of  their  participation. 

Instrument. 

All  participants  were  given  the  same  two-page  scenario  description.  The 
description  stated  that  the  primary  weapon  system  outcome  of  concern  was  Operational 
Availability.  The  description  also  pointed  out  that  the  component  which  had  been 
outsourced  to  the  vendor  was  only  1  component  of  many  which  determined  operational 
availability  for  the  weapon  system;  and  depot-level  maintenance  of  that  component  was 
only  one  logistical  element  of  many  which  determined  the  performance  of  that 
component  in  the  aircraft.  See  Figure  1 . 
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Figure  1.  Graphic  Given  to  Participants  to  Support  Description  of  Depot-Level 
Repair  within  System-Level  (Aircraft)  Availability 


The  three  forms  of  the  instrument  differed  in  the  data  that  was  presented  to  the 
participants.  In  the  base  case,  participants  were  given  only  data  on  how  many  weapon 
systems  (aircraft)  were  down  due  to  failure  of  the  component.  As  aircraft  availability 
was  stated  to  be  the  outcome  of  concern,  the  data  provided  was  a  good  outcome 
surrogate.  Weekly  data  were  provided  from  a  two-year  period  (prior  year  and  contract 
year),  and  the  average  number  of  aircraft  down  due  to  BQV  failure  was  shown  to  have 
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increased  slightly  during  the  contract  year  from  the  prior  year  (i.e.,  performance 
worsened). 

In  the  second  form  of  the  instrument,  in  addition  to  the  outcome  information, 
participants  were  also  given  data  from  process  measures  which  only  partially  correlated 
with  the  outcome  of  concern.  In  particular,  participants  were  given  weekly  data  from  a 
two-year  period  (prior  year  and  contract  year)  on  the  average  mean  time  between 
failures  for  engines  in  a  given  week,  average  repair  cycle  time,  and  average  days  to 
supply  backorders.  In  each  case,  the  impact  on  system  performance  cannot  be 
determined  directly  from  the  numbers.  However,  in  each  case,  performance  on  these 
process  measures  improved  slightly  from  the  prior  year  to  the  contract  year. 

In  the  third  form  of  the  instrument,  participants  were  not  given  the  process 
measures,  but  attention  was  drawn  to  the  variability  of  the  outcome  (number  of  aircraft 
down  due  to  component  failure).  The  only  additional  piece  of  information  given  to  the 
participants  on  the  third  form  was  the  standard  deviation  of  the  outcome  for  each  year. 
Variance  in  the  contract  year  had  been  reduced  by  a  factor  of  four  compared  to  the  prior 
year,  from  14.3%  standard  deviation  in  aircraft  availability  to  6.0%  standard  deviation  in 
aircraft  availability.  In  other  words,  while  average  performance  was  very  slightly  worse 
in  the  contract  year  compared  to  the  prior  year,  performance  was  far  more  reliable  in  the 
contract  year.  In  the  base  case,  the  participants  were  given  weekly  data  from  2  years 
(52  x  2  numbers),  but  the  standard  deviation  was  not  given.  The  participants  in  the 
base  case  could  have  calculated  the  number  themselves,  or  simply  observed  the  large 
decrease  in  variability  by  carefully  checking  the  data;  however,  they  were  not  directed  to 
do  so.  On  the  third  form  of  the  instrument,  the  standard  deviation  of  outcomes  was 
given  in  order  to  draw  attention  to  the  reduction  in  variability. 

Every  participant  answered  the  same  seven  questions.  There  were  6  items  in 
the  survey  intended  to  provide  a  global  assessment  of  vendor  performance  (this  was 
the  scale  being  piloted)  and  a  seventh  item  asking  participants  to  set  a  performance 
award  for  the  vendor  (0-5  percent  of  cost  bonus).  The  seventh  item  was  intended  to 
provide  a  check  of  convergent  validity  for  the  first  six  items.  The  six  items  in  the  vendor 
performance  scale  are  given  in  the  Appendix. 


15 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


16 


Analyses. 

Cronbach’s  alpha  was  used  to  assess  internal  reliability  of  the  six-item  scale. 
Independent  sample  t-tests  were  used  to  assess  differences  between  participants’ 
responses  when  given  one  of  the  two  ’’treatment”  forms  (containing  additional  data)  and 
participants’  responses  when  given  the  base-case  form.  Congruence  between  outcome 
variables  (six-item  scale  or  percentage  award)  was  assessed  qualitatively  by  examining 
descriptive  statistics  and  the  outcome  of  t-tests  run  using  each  of  the  outcome 
variables. 

Results. 

Descriptive  statistics  (mean  and  standard  deviation)  for  each  of  the  two 
dependent  variables  are  given  in  Table  1  for  all  three  participant  groups. 

Table  1.  Pilot  Results  (Mean,  Standard  Deviation) 


Group 

Outcome 

Only 

(base  case) 

Process 

Measures 

Given 

Std.  Dev. 
Given 

6-Item 

Performance 

Scale 

14.3,  3.7 

17.4,4.4 

16.6,  3.8 

Percent 

Award 

0.8,  1.2 

0.5,  0.8 

1.5,  1.2 

The  six-item  vendor  performance  scale  exhibited  reasonable  internal  reliability  on 
this  sample  (a  =  .791,  N  =  61). 
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Independent  sample  t-tests  showed  mixed  support  for  Hypothesis  2.  Although 
participants’  performance  evaluation  increased  when  given  data  on  process  measures  (t 
=  2. 1 1 6,  p  =  .022),  their  percentage  award  to  the  vendor  did  not  increase  (t  =  -0.854,  p  = 
.20).  The  failure  to  find  the  predicted  result  for  the  percentage  award  result  may  be 
partially  be  due  to  a  combination  of  high  variance  (the  coefficient  of  variation  is  greater 
than  one  for  both  groups,  and  the  t-test  is  not  significant)  and  a  truncation  effect  (a  large 
number  of  participants  gave  zero  award  in  both  groups). 

Independent  sample  t-tests  showed  strong  support  for  Hypothesis  4.  On  both 
the  performance  evaluation  scale  (t  =  1 .782,  p  =  .043)  and  the  percentage  award  (t  = 
1.586,  p  =  .062),  evaluations  were  significantly  higher  when  the  difference  in  variance 
was  made  plain  by  printing  the  standard  deviation  with  the  data. 

The  results  show,  at  best,  mixed  support  for  the  convergent  validity  of  the 
performance-evaluation  scale  and  percentage-bonus  awards.  This  may  be  due  to 
technical  issues  with  the  percentage-award  scale  (noted  above)  or  it  may  be  that 
participants  viewed  performance  evaluation  (as  measured  by  the  six-item  scale)  as 
different  in  substance  from  performance  awards. 
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Limitations,  Discussion  and  Conclusion 


The  primary  purpose  of  the  pilot  data  collection  was  to  validate  a  scale  for  use  in 
further  testing;  however,  since  we  have  reported  results  from  this  pilot  experiment,  we 
should  also  make  clear  a  number  of  limitations  to  interpreting  the  results.  First,  a 
number  of  controls  and  manipulation  checks  were  omitted  in  the  pilot  experiment.  For 
example,  while  we  tried  to  make  it  clear  that  system  availability  was  the  important 
weapon  system  outcome  to  be  assessed  in  the  pilot,  we  did  not  perform  a  manipulation 
check  to  make  sure  that  participants  understood  this.  Also,  while  demographic  data 
were  gathered,  no  attempt  was  made  on  these  pilot  data  to  ascertain  if  these  variables 
explained  a  significant  amount  of  variance  in  the  dependent  variables.  Future  work,  for 
example,  should  treat  experience  with  contracting  and  logistics  as  a  control  variable. 
Finally,  while  we  have  been  able  to  report  preliminary  support  for  two  of  our  hypotheses, 
the  other  two  hypotheses  could  not  be  tested  with  the  pilot  instrument. 

In  our  review  of  the  literature,  we  have  drawn  a  connection  between  the  intent  of 
performance-based  service  contracting  and  PBL  on  the  one  hand,  and  the  literature  on 
biases  and  heuristics  on  the  other.  We  have  shown  that  the  core  intentions  of 
performance-based  contracting — evaluating  performance  based  on  outcomes  rather 
than  processes — is  itself  held  in  question  by  decision-making  researchers. 

We  have  seen  from  a  review  of  the  logistics  measurement  literature  that  logistic 
services  themselves  are  hard  to  value  and  evaluate,  in  part  because  outcomes  are 
difficult  to  quantify.  In  such  a  circumstance,  behavioral  decision  research  suggests  that 
decision  makers  will  use  outcome  surrogates,  including  inputs,  to  value  outcomes.  If 
these  strategies  are  also  used  in  PBL  services,  their  usage  would  indicate  a  greater 
need  for  guidance  in  the  outcome  measures  that  should  be  used  for  PBL  contract 
management. 

Much  of  the  earlier  guidance  on  PBL  insisted  upon  the  idea  that  there  was  not 
“one  best  way”  to  implement  PBL,  and  that  measurement,  consequently,  would  also 
need  to  vary  from  contract  to  contract.  While  more  specific  guidance  has  been  given  in 
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the  last  year  clarifying  that  the  important  performance  outcomes  under  PBL  are  at  the 
system  level,  to  our  knowledge  little  specific  guidance  has  been  given  on  the  precise 
mechanisms  to  be  used  to  connect  component-level  performance  with  system-level 
outcomes.  Under  such  ambiguity,  we  have  seen  that  behavioral  decision  making 
researchers  predict  that  decision  makers  will  seek  out  and  use  process  measures  that 
are  non-diagnostic  (not  relevant  to  the  evaluation  task)  or  only  partially  diagnostic  of 
outcomes.  Our  pilot  data  provided  mixed  and  limited  support  for  the  idea  that  decision 
makers  will  use  process  measures,  even  when  told  that  outcomes  are  more  valued,  and 
even  when  given  clear  outcome  measures.  Whether  decision  makers  will  actually  seek 
out  such  process  data  (Hypothesis  1)  remains  untested. 

Finally,  in  reviewing  the  literature  on  PBL,  we  have  noted  that  one  of  the  primary 
intentions  of  the  initiative  is  to  outsource  process  risk.  We  have  noted  that  process  risk 
connects  to  outcome  performance  through  outcome  variance,  and  that  it  is  possible  to 
evaluate  the  success  of  process  risk  transfer  without  measuring  processes  directly,  but 
merely  through  a  direct  evaluation  of  outcome  variance.  However,  we  have  also  noted 
that  one  of  the  core  lessons  of  the  behavioral  decision  making  literature  is  that 
managers  are  especially  poor  intuitive  statisticians,  rely  heavily  on  heuristics  such  as 
representativeness  to  assess  probability,  and  are  subject  to  the  influence  of  a  number  of 
biases  (such  as  base  rate  neglect).  Our  pilot  data  have  provided  limited  support  for  the 
idea  that  decision  makers  will  simply  neglect  variance  information  unless  it  is  made 
salient.  If  decision  makers  under  PBL  are  subject  to  the  same  limitations — if  further 
support  is  found  for  our  Hypothesis  4 — that  data  indicates  the  need  for  the  DoD  to 
develop  specific  guidance  with  regard  to  risk  measurement  and  valuation  under  PBL. 
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Appendix  1.  Component  Logistic  Service  Vendor 
Performance  Scale 


(Note:  the  BQV  is  a  fictitious  component  in  the  H-80,  a  fictitious  aircraft.  The  contract 
year  reviewed  was  2003.  All  items  were  measured  on  a  five-point  scale,  in  which  one 
was  “strongly  disagree,”  three  was  “neutral,”  and  five  was  “strongly  agree.”) 


1 .  Logistical  support  of  the  BQV  for  the  H-80  was  good  in  2003. 

2.  The  vendor  improved  logistical  support  outcomes  of  the  BQV  in  2003  to  above 
performance  in  2002. 

3.  The  BQV  vendor  provided  better  response  in  support  of  the  H-80  in  2003  than  in 
2002. 

4.  The  vendor  improved  performance  outcomes  of  the  BQV  in  2003  to  above 
performance  in  2002. 

5.  The  vendor  improved  service  levels  in  2003  to  above  service  levels  in  2002. 

6.  Squadrons  were  less  likely  to  be  short  a  BQV  for  an  H-80  in  2003  than  they  were 
in  2002. 
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FY  2005  Sponsored  Acquisition  Research  Products 


Sponsored  Report  Series 

NPS-LM-05-016  Impact  of  Diffusion  and  Variability  on 


NPS-PM-05-017 

Vendor  Performance  Evaluation 

September  2005 

Department  of  the  Navy  Procurement  Metrics  Evaluation 

June  2005 

NPS-CM-05-015 

Joint  Contingency  Contracting 

June  2005 

NPS-AM-05-014 

Innovations  in  Funding  the  Maritime  Prepositioning 

Ships  Program  -  A  Case  Analysis 

June  2005 

NPS-CM-05-013 

Update  of  the  Navy  Contract  Writing  Guide  Phase  III 

June  2005 

NPS-PM-05-012 

The  Raven  Small  Unmanned  Aerial  Vehicle  (SUAV),  Investigating 
Potential  Dichotomies  between  Doctrine  and  Practice 

June  2005 

NPS-AM-05-01 1 

Spiral  Development:  A  Perspective 

June  2005 

NPS-FM-05-009 

Budgeting  for  National  Defense  Acquisition:  Assessing  System 
Linkage  and  the  Impact  of  Transformation 

June  2005 

NPS-LM-05-008 

A  Decision  Support  Model  for  Valuing  Proposed  Improvements  in 
Component  Reliability 

June  2005 

NPS-PM-05-007 

Determining  the  Best  Loci  of  Knowledge,  Responsibilities  and 
Decision  Rights  in  Major  Acquisition  Organizations 

June  2005 

NPS-AM-05-006 

Navy  Acquisition  Via  Leasing:  Policy,  Politics,  and  Polemics  with 
the  Maritime  Prepositioned  Ships 

April  2005 

NPS-CM-05-003 

Commodity  Sourcing  Strategies:  Supply  Management  in  Action 

January  2005 

Working  Paper  Series 

NPS-AM-05-010  From  Market  to  Clan:  How  Organizational  Control  Affects  Trust  in 
Defense  Acquisition 
June  2005 
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NPS-AM-05-005 

Cost  as  an  Independent  Variable  (CAIV):  Front-End  Approaches  to 
Achieve  Reduction  in  Total  Ownership  Cost 

June  2005 

NPS-AM-05-002 

The  Yoder  Three-Tier  Model  for  Optimizing  Contingency 
Contracting  Planning  and  Execution 

December  2004 

NPS-AM-05-001 

Engagement  versus  Disengagement: 

How  Structural  &  Commercially-Based  Regulatory  Changes  have 
Increased  Government  Risks  in  Federal  Acquisitions 

November  2004 

Acquisition  Symposium  Proceedings 

NPS-AM-05-004  Acquisition  Research:  The  Foundation  for  Innovation 
May  2005 

FY  2004  Sponsored  Acquisition  Research  Products 

Sponsored  Report  Series 

NPS-CM-04-019  Contractor  Past  Performance  Information  (PPI)  In  Source 


NPS-LM-04-014 

Selection:  A  comparison  Study  of  Public  and  Private  Sector 
December  2004 

Optimizing  Phalanx  Weapon  System  Life-Cycle  Support 

October  2004 

NPS-AM-04-013 

Business  Case  Analysis  and  Contractor  vs.  Organic  Support: 

A  First-Principles  View 

September  2004 

NPS-CM-04-006 

Measurement  Issues  in  Performance  Based  Logistics 

June  2004 

NPS-CM-04-004 

Update  of  the  Navy  Contract  Writing,  Phase  II 

June  2004 

NPS-CM-04-001 

Update  of  the  Navy  Contract  Writing,  Phase  1 

December  2003 

NPS-CM-04-002 

Marine  Corps  Contingency  Contracting  MCI 

December  2003 

Working  Paper  Series 

NPS-PM-04-017  The  New  Joint  Capabilities  Integration  Development  System 


NPS-CM-04-016 

(JCIDS)  and  Its  Potential  Impacts  upon  Defense  Program 
Managers 

December  2004 

An  Analysis  of  Industry’s  Perspective  on  the  Recent  Changes  to 
Circular  A-76 

October  2004 
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NPS-CM-04-012 

Air  Force  Commodity  Councils:  Leveraging  the  Power  of 
Procurement 

September  2004 

NPS-CM-04-01 1 

Using  Metrics  to  Manage  Contractor  Performance 

September  2004 

NPS-LM-04-009 

Improving  the  Management  of  Reliability 

August  2004 

NPS-AM-04-007 

The  Impact  of  Software  Support  on  System  Total  Ownership  Cost 
July  2004 

NPS-LM-04-003 

Enablers  to  Ensure  a  Successful  Force  Centric  Logistics  Enterprise 

April  2004 

Acquisition  Case  Series 

NPS-CM-04-008  Privatization  of  the  Naval  Air  Warfare  Center  Aircraft  Division, 
Indianapolis 
July  2004 

NPS-PM-04-010  The  Army  Seeks  a  World  Class  Logistics  Modernization  Program 
June  2004 

Acquisition  Symposium  Proceedings 

NPS-AM-04-005  Charting  a  Course  for  Change:  Acquisition  Theory  and  Practice  for 
a  Transforming  Defense 
May  2004 

FY  2003  Sponsored  Acquisition  Research  Products 

Sponsored  Report  Series 


NPS-AM-03-003 

Centralized  Control  of  Defense  Acquisition  Programs:  A 
Comparative  Review  of  the  Framework  from  1987  -  2003 
September  2003 

NPS-AM-03-004 

Reduction  of  Total  Ownership  Cost 

September  2003 

NPS-CM-03-006 

Auto-Redact  Toolset  for  Department  of  Defense  Contracts 

September  2003 

Working  Paper  Series 

NPS-CM-03-002  Transformation  in  DOD  Contract  Closeout 
June  2003 
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Acquisition  Case  Series 

NPS-CM-03-005  Contract  Closeout  (A) 

September  2003 

Other  Sponsored  Research 

NPS-CM-03-001  Transformation  in  DOD  Contract  Closeout 
MBA  Professional  Report 
June  2003 

Copies  of  the  Acquisition  Sponsored  Research  Reports  may  be  printed  from  our 
website  www.nps.navy.mil/qsbpp/acqn/publications 
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